Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus that generates a combined image by performing gamma processing and combination processing for a plurality of images obtained by capturing images of the same image capturing-target scene under different exposure conditions and includes: an acquisition unit configured to acquire adjustment parameters that adjust gamma characteristics applied to the gamma processing in accordance with a dynamic range of the image capturing-target scene; a gamma processing unit configured to perform the gamma processing to which adjustment parameters acquired by the acquisition unit have been applied for the plurality of images; and a combination unit configured to generate the combined image by performing the combination processing for the plurality of images for which the gamma processing has been performed by the gamma processing unit.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique to generate a high dynamicrange image.

Description of the Related Art

In general, the dynamic range of an image sensor, such as a CCD and aCMOS, used in an image capturing apparatus, such as a digital camera anda digital video camera, is narrow compared to the dynamic range in thenatural world. Because of this, in the case where the image of an imagecapturing-target scene having a wide dynamic range (referred to as ahigh dynamic range and in the present specification, abbreviated to“HDR”) is captured by a common method, blocked up shadows, blown outhighlights, and so on occur. As a technique to capture the image of ascene having such a wide dynamic range (in the present specification,referred to as an HDR scene), there is a method of acquiring the imageof an HDR scene by capturing a plurality of images with differentexposure times and combining the obtained plurality of images. That is,level matching and gamma processing are performed for a plurality ofimages captured with different exposure times based on the exposuretimes. Then, by selecting an optimum pixel for each area from among theimages for which the level matching and gamma processing have beenperformed, the image of an HDR scene is acquired. Hereinafter, such acombination method is referred to as HDR combination.

However, in the case where the HDR combination is applied to a scenehaving a dynamic range different from the supposed one, there is apossibility that an image that extremely lacks contrast is generated.Consequently, it is necessary to acquire images under an exposurecondition optimum for the dynamic range of the scene and to performoptimum HDR combination. The image capturing apparatus of JapanesePatent Laid-Open No. 2002-135648 determines the exposure conditionoptimum for a scene and performs image capturing and combination bycalculating the dynamic range of the scene from histograms ofpreliminary captured images and determining the exposure condition atthe time of the main image capturing. Further, in the image outputtingmethod of Japanese Patent Laid-Open No. 2004-120205, the image optimumfor the dynamic range of a scene is generated by calculating the dynamicrange of the scene for the image after the HDR combination by thehistogram analysis and performing the gamma correction again for theimage after the combination based on the results of the calculation.

However, in the image capturing apparatus of Japanese Patent Laid-OpenNo. 2002-135648, the exposure condition at the time of the main imagecapturing is changed in accordance with the dynamic range of the scenecalculated from the histograms of the preliminary captured images.Because of this, it is necessary to prepare parameters required toperform the HDR combination for each dynamic range of the scene, andtherefore, a very large number of combinations of parameters will benecessary.

Further, with the image outputting method of Japanese Patent Laid-OpenNo. 2004-120205, the gamma correction is performed again for the imageafter the HDR combination, and therefore, unless a sufficient number ofbits is secured, a pseudo contour occurs in the image after the gammacorrection. That is, in the case where the gamma correction is performedagain for the image after the HDR combination, the deterioration of theimage quality (bit dropout) due to the gamma correction performed two ormore times, or the increase in the data amount due to the bit expansionwill be problematic.

Consequently, the present invention provides an image processingapparatus capable of generating a combined image of high image qualityby HDR combination with a small number of bits without requiring thepreparation of a very large number of parameters for scenes having avariety of dynamic ranges.

SUMMARY OF THE INVENTION

The image processing apparatus according to the present invention is animage processing apparatus that generates a combined image by performinggamma processing and combination processing for a plurality of imagesobtained by capturing images of the same image capturing-target sceneunder different exposure conditions, and the image processing apparatusincludes: an acquisition unit configured to acquire adjustmentparameters that adjust gamma characteristics applied to the gammaprocessing in accordance with a dynamic range of the imagecapturing-target scene; a gamma processing unit configured to performthe gamma processing to which the adjustment parameters acquired by theacquisition unit have been applied for the plurality of images; and acombination unit configured to generate the combined image by performingthe combination processing for the plurality of images for which thegamma processing has been performed by the gamma processing unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A and FIG. 1B are each a diagram showing an external appearance ofan image capturing apparatus in a first embodiment;

FIG. 2 is a block diagram showing an internal configuration of the imagecapturing apparatus in the first embodiment;

FIG. 3 is a flowchart of entire processing of the image capturingapparatus in the first embodiment;

FIG. 4 is a diagram showing details of image processing performed by animage processing unit in the first embodiment;

FIG. 5 is a diagram showing a relationship between a luminance value mapand a composite coefficient value used in HDR image combinationprocessing of the first embodiment;

FIG. 6A to FIG. 6D are diagrams for explaining adjustment parametercalculation processing of the first embodiment;

FIG. 7 is a flowchart of entire processing of an image capturingapparatus in a second embodiment; and

FIG. 8 is a diagram showing details of image processing performed by animage processing unit in the second embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following, embodiments of the present invention are explainedwith reference to the drawings. However, the following embodiments donot limit the present invention and all the combinations of featuresexplained below are not necessarily indispensable to solve the problemof the present invention. In each drawing referred to in the followingexplanation, as a principle, the same symbol is attached to the elementequivalent to that in another drawing.

First Embodiment

In the present embodiment, first, after performing developmentprocessing, such as white balance, demosaicking, and gamma processing,for a plurality of images captured under different exposure conditions,HDR combination is performed. Next, histogram analysis is performed forthe image for which the HDR combination has been performed and based onthe results of the analysis, adjustment parameters to adapt the gammaprocessing to the dynamic range of an image capturing-target scene arecalculated. Then, the main gamma processing to which the calculatedadjustment parameters have been applied is performed for the pluralityof captured images.

FIG. 1A and FIG. 1B are each a diagram showing the external appearanceof an image capturing apparatus that can be adapted to the presentembodiment. In the present embodiment, as an example configuring animage capturing apparatus, a configuration example of a digital camerais illustrated. Although description is given to a digital camera, butthe configuration example is not limited to this. For example, theconfiguration example may be an information processing apparatus ordevice, such as a mobile telephone, a tablet device, and a personalcomputer, or the image capturing apparatus may be configured as an imagecapturing apparatus of a camera-attached mobile telephone and the like.

FIG. 1A shows an external appearance of the front side of an imagecapturing apparatus 101 and FIG. 1B shows the external appearance of therear side. The image capturing apparatus 101 includes an optical unit102, an image capturing button 103, a display unit 104, and an operationbutton 105. The optical unit 102 includes a zoom lens, a focus lens, acamera-shake correction lens, an aperture stop, and a shutter andcollects light information on a subject. The image capturing button 103is a button for a user to give instructions to start image capturing tothe image capturing apparatus 101. As the display unit 104, a liquidcrystal display or the like is used and the display unit 104 displays animage processed by the image capturing apparatus 101, various kinds ofdata, and so on. The operation button 105 is a button for a user tospecify parameters of exposure conditions and the like to the imagecapturing apparatus 101.

FIG. 2 is a block diagram showing an internal configuration of the imagecapturing apparatus 101 in the present embodiment. A color image sensingelement unit 201 is an element that converts light information collectedby the optical unit 102 into a current value and acquires colorinformation by being combined with a color filter or the like.

A CPU 202 is a processor that centralizedly controls each configurationwithin the image capturing apparatus 101. A ROM (Read Only Memory) 203is a memory that stores a program group executed by the CPU 202. A RAM(Random Access Memory) 204 is a memory that functions as a main memory,a work area, and so on, of the CPU 202. The CPU 202 performs processingin accordance with programs stored in the ROM (Read Only Memory) 203 orprograms loaded onto the RAM 204.

An image capturing system control unit 205 performs control specified bythe CPU 202, such as focusing, shutter opening, and aperture stopadjustment. A control unit 206 performs control to start and end theimage capturing operation based on instructions input by a user via theimage capturing button 103 or the operation button 105. A charactergeneration unit 207 generates characters, graphics, and so on.

An A/D conversion unit 208 converts a light quantity of a subjectdetected by the optical unit 102 into a digital signal value andgenerates a RAW image. In the present embodiment, explanation is givenby an example in which the RAW image is an image having only one colorof R, G, and B in each pixel.

An image processing unit 209 performs image processing including whitebalance processing, demosaicking processing, gamma processing, and HDRcombination for the above-described RAW image. An encoder unit 210converts an image, which is the results of the processing by the imageprocessing unit 209, into a file format, such as a Jpeg.

A media I/F 211 is an interface for performing transmission andreception of image data with a PC/media 213 (e.g., hard disk, memorycard, CF card, SD card, and so on). A system bus 212 is a bus forperforming transmission and reception of various kinds of data with eachconfiguration within the image capturing apparatus 101.

FIG. 3 is a flowchart of entire processing of the image capturingapparatus 101 of the present embodiment. FIG. 4 is a diagram showingdetails of image processing performed by the image processing unit 209.In the following, with reference to FIG. 3 and FIG. 4, detailedexplanation is given.

At step S301, a user sets exposure conditions, such as an f-stop of alens and an ISO speed, via the operation button 105. The exposure timeis not changed for each scene, but a condition prepared in advance isset. In the present embodiment, explanation is given by taking imagecapturing of an HDR scene with two kinds of exposure time as an example.In the following, an image captured with the relatively short exposuretime of the two kinds of exposure time is called a short-time exposureimage and an image captured with the relatively long exposure time iscalled a long-time exposure image.

At step S302, the control unit 206 determines whether or not the imagecapturing button 103 has been pressed down. In the case where the imagecapturing button 103 has been pressed down, the processing advances tostep S303 and in the case where the image capturing button 103 has notbeen pressed down, the processing returns to step S301.

At step S303, the control unit 206 sets the focus position by autofocusing and sets the exposure conditions (image capturing parameters)set by a user at step S301 to the image capturing system control unit205 and instructs the image capturing system control unit 205 to startthe image capturing operation. Then, the image capturing system controlunit 205 performs control so that image capturing is performed aplurality of times with different exposure times (exposure bracket imagecapturing) based on the set exposure conditions. Specifically, the imagecapturing system control unit 205 acquires the light quantity of asubject by driving the optical unit 102 and further, the A/D conversionunit 208 acquires a RAW image by converting the acquired light quantityof a subject into a digital signal value. In the present embodiment, asthe RAW image, a long-time exposure image 41 captured with a longexposure time (e.g., 1/120 sec) and a short-time exposure image capturedwith a short exposure time (e.g., 1/2000 sec) for the same scene areacquired and input to the image processing unit 209.

At step S304, the image processing unit 209 performs white balanceprocessing 401 for the input long-time exposure image 41 and short-timeexposure image 42, respectively. At step S305, the image processing unit209 performs demosaicking processing 402 for the images, respectively,for which the white balance processing has been performed at step S304.Due to this, a long-time exposure image 43 and a short-time exposureimage 44 are acquired.

At step S306, the image processing unit 209 performs long-timepreliminary gamma processing 403 and short-time preliminary gammaprocessing 404 for the long-time exposure image 43 and the short-timeexposure image 44, respectively, acquired at step S305. Details of thepreliminary gamma processing at this step will be described later.

At step S307, the image processing unit 209 performs preliminary HDRcombination processing 405 for the image for which the preliminary gammaprocessing has been performed at step S306. In the following, the imageas the results of the preliminary HDR combination processing is called apreliminary HDR combined image. Details of the preliminary HDRcombination processing at this step will be described later.

At step S308, the image processing unit 209 performs histogramcalculation 406 for the preliminary HDR combined image as the results ofthe processing at step S307. The histogram calculation is performed foreach of a luminance value Y and R, G, and B values of the preliminaryHDR combined image.

At step S309, the image processing unit 209 performs adjustmentparameter calculation (407) based on the histograms calculated at stepS308. That is, the image processing unit 209 calculates adjustmentparameters 45 that adjust the gamma characteristics applied to the gammaprocessing in accordance with the dynamic range of a scene. Details ofthe adjustment parameter calculation at this step will be describedlater.

At step 310, the image processing unit 209 performs the main gammaprocessing for the long-time exposure image 43 and the short-timeexposure image 44, respectively, acquired at step S305 based on theadjustment parameters 45 calculated at step S309. That is, long-timegamma processing 408 and short-time gamma processing 409 are performed.

At step S311, the image processing unit 209 performs HDR combinationprocessing 410 for the image for which the main gamma processing hasbeen performed at step S310 and generates a combined image 46. Detailsof the HDR combination processing at this step will be described later.

At step S312, the image processing unit 209 outputs the combined image46 generated at step S311 to the encoder unit 210, to the PC/media 213via the media I/F 211, and so on.

In the present embodiment, the image processing unit 209 calculates theluminance value Y and color difference values U and V from the R, G, andB values of an input image. That is, the image processing unit 209performs color space conversion processing from RGB→YUV. Then, the imageprocessing unit 209 calculates the luminance value Y and the colordifference values U and V after the gamma correction by performing loggamma correction on the YUV color space. Further, the image processingunit 209 calculates the R, G, and B values after the gamma correctionfrom the luminance value Y and the color difference values U and V afterthe gamma correction.

<Preliminary Gamma Processing>

The preliminary gamma processing of the present embodiment is gammaprocessing based on predetermined gamma characteristics. In thefollowing, as the preliminary gamma processing at step S306, thelong-time preliminary gamma processing 403 performed for the long-timeexposure image 43 and the short-time preliminary gamma processing 404performed for the short-time exposure image 44 are explained in detail.

For the long-time exposure image 43, the long-time preliminary gammaprocessing 403 is performed by expression (1) below. Here, it is assumedthat the R, G, and B values of the long-time exposure image 43 areR_(long), G_(long), and B_(long), respectively, and the R, G, and Bvalues after the gamma correction are R_(long)′, G_(long)′, andB_(long)′, respectively. Further, it is assumed that the luminance valueY and the color difference values U and V of the long-time exposureimage 43 are Y_(long), U_(long), and V_(long), respectively, and theluminance value Y and the color difference values U and V after thegamma correction are Y_(long)′, U_(long)′, and V_(long)′, respectively.Furthermore, Max and O_(max) are gamma correction coefficients set inadvance and Max is set to the upper limit value of the luminance value Ybefore the preliminary gamma correction and O_(max) is set to the upperlimit value of the luminance value Y after the preliminary gammacorrection, respectively. For example, it is possible to set Max to2̂²³−1 and O_(max) to 1,023 in the case where a 10-bit output isproduced.

                                expression  (1)Y_(long) = 0.257 * R_(long) + 0.504 * G_(long) + 0.114 * B_(long)U_(long) = −0.169 * R_(long) − 0.331 * G_(long) + 0.500 * B_(long)V_(long) = 0.500 * R_(long) − 0.419 * G_(long) − 0.081 * B_(long)$Y_{long}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( Y_{long} \right)}}$$U_{long}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( U_{long} \right)}}$$V_{long}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( V_{long} \right)}}$R_(long)^(′) = 1.00 * Y_(long)^(′) + 1.402 * V_(long)^(′)G_(long)^(′) = 1.00 * Y_(long)^(′) − 0.344 * U_(long)^(′) + 1.402 * V_(long)^(′)B_(long)^(′) = 1.00 * Y_(long)^(′) + 1.772 * U_(long)^(′)

Further, for the short-time exposure image 44, the short-timepreliminary gamma processing 404 is performed by expression (2) below.Here, it is assumed that the R, G, and B values of the short-timeexposure image 44 are R_(short), G_(short), and B_(short), respectively,and the R, G, and B values after the gamma correction are R_(short)′,G_(short)′, and B_(short)′, respectively. Further, it is assumed thatthe luminance value Y and the color difference values U and V of theshort-time exposure image 44 are Y_(short), U_(short), and V_(short),respectively, and the luminance value Y and the color difference valuesU and V after the gamma correction are Y_(short)′, U_(short)′ andV_(short)′, respectively. Furthermore, it is assumed that the exposuretime of the short-time exposure image 44 is t_(short) and the exposuretime of the long-time exposure image 43 is t_(long).

                                expression  (2)Y_(short) = 0.257 * R_(short) + 0.504 * G_(short) + 0.114 * B_(short)U_(short) = −0.169 * R_(short) − 0.331 * G_(short) + 0.500 * B_(short)V_(short) = 0.500 * R_(short) − 0.419 * G_(short) − 0.081 * B_(short)$Y_{short}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( {\frac{t_{long}}{t_{short}}Y_{short}} \right)}}$$U_{short}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( {\frac{t_{long}}{t_{short}}U_{short}} \right)}}$$V_{short}^{\prime} = {\frac{O_{\max}}{\log ({Max})}{\log \left( {\frac{t_{long}}{t_{short}}V_{short}} \right)}}$R_(short)^(′) = 1.00 * Y_(short)^(′) + 1.402 * V_(short)^(′)G_(short)^(′) = 1.00 * Y_(short)^(′) − 0.344 * U_(short)^(′) + 1.402 * V_(short)^(′)B_(short)^(′) = 1.00 * Y_(short)^(′) + 1.772 * U_(short)^(′)

<Preliminary HDR Combination Processing>

Details of the preliminary HDR combination processing 405 at step S307are explained with reference to FIG. 5. For the long-time exposure imageand the short-time exposure image for which the preliminary gammaprocessing has been performed at step S306, the preliminary HDRcombination processing 405 is performed by expression (3) below.

Out(x,y)=A×I _(short)(x,y)+(1−A)×I _(long)(x,y)  expression (3)

Here, x and y indicate a pixel position. Out (x, y) is a pixel value (R,G, and B values) stored at the pixel position (x, y) of the output imageafter the combination. I_(short) (x, y) is a pixel value stored at thepixel position (x, y) of the short-time exposure image after thepreliminary gamma processing has been applied, i.e., R_(short)′(x, y),G_(short)′(x, y), and B_(short)′(x, y). I_(long) (x, y) is a pixel valuestored at the pixel position (x, y) of the long-time exposure imageafter the preliminary gamma processing has been applied, i.e.,R_(long)′(x, y), G_(long)′(x, y), and B_(long)′(x, y). Further, A is acomposite coefficient value stored at the pixel position (x, y) of acomposite coefficient map. The composite coefficient map is a map thatstores the composite coefficient value of the short-time exposure imageused in the preliminary HDR combination processing for each pixel. Inthe following, a generation method of the composite coefficient map isexplained.

FIG. 5 is a graph to determine the composite coefficient value A of thecomposite coefficient map for the luminance value. The compositecoefficient map in the present embodiment is a map that stores compositecoefficient values of the short-time exposure image and generated basedon the short-time exposure image after the preliminary gamma processinghas been applied and the graph in FIG. 5. Specifically, from a pixelvalue I_(short) (R, G, B values) of the short-time exposure image, theluminance value of each pixel is calculated and smoothing processing isperformed for the calculated luminance value. To the smoothingprocessing, it is possible to apply, for example, a 5×5 Gaussian filter.Then, for the luminance value after the smoothing processing, thecomposite coefficient value A of the composite coefficient map isdetermined based on the graph in FIG. 5. As shown in FIG. 5, a thresholdvalue Th1 and a threshold value Th2 are set and in the case where theluminance value is smaller than or equal to the threshold value Th1, thecomposite coefficient value A is determined to be 0 and in the casewhere the luminance value is larger than or equal to the threshold valueTh2, the composite coefficient value A is determined to be 1. Further,in the case where the luminance value is between the threshold value Th1and the threshold value Th2, as the composite coefficient value A, theresults of performing linear interpolation are used. As described above,the composite coefficient value A at each pixel is determined and thecomposite coefficient map is generated.

In the present embodiment, explanation is given on the assumption thatthe composite coefficient value A is determined by having the twothreshold values and performing interpolation, but the determinationmethod of the composite coefficient value A is not limited to this. Forexample, it may also be possible to determine the composite coefficientvalue A by having three or more threshold values and performinginterpolation, or to determine the composite coefficient value A byusing another expression.

<Adjustment Parameter Calculation>

Details of the adjustment parameter calculation at step S309 areexplained with reference to FIG. 6A to FIG. 6D. FIG. 6A to FIG. 6D arediagrams for explaining adjustment parameter calculation processing ofthe present embodiment and show an example in which histograms of theluminance value Y and the R, G, and B values are calculated for apreliminary combined image. In the case where the maximum values and theminimum values of each histogram are taken to be Ymin, Ymax, Rmin, Rmax,Gmin, Gmax, Bmin and Bmax, adjustment parameters Ay and By for theluminance value and adjustment parameters Ac and Bc for the colordifference value are calculated by expression (4) below.

$\begin{matrix}\left\{ {\begin{matrix}{{Ay} = \frac{M_{out}}{{Y\; \max} - {Y\; \min}}} \\{{By} = {{{Ay} \cdot Y}\; \min}}\end{matrix}\left\{ \begin{matrix}{{Ac} = \frac{M_{out}}{{C\; \max} - {C\; \min}}} \\{{Bc} = {{{Ac} \cdot C}\; \min}} \\{{C\; \min} = {\min \left( {{R\; \min},{G\; \min},{B\; \min}} \right)}} \\{{C\; \max} = {\max \left( {{R\; \max},{G\; \max},{B\; \max}} \right)}}\end{matrix} \right.} \right. & {{expression}\mspace{14mu} (4)}\end{matrix}$

Here, M_(out) is a coefficient set in advance. For example, it ispossible to set M_(out) to 1,023 in the case where the number of outputbits is ten. As is expressed by expression (4), as the differencebetween Ymin and Ymax becomes large, Ay becomes small. As Ymin becomessmall, By becomes small. Further, as the difference between Cmin andCmax becomes large, Ac becomes small. As Cmin becomes small, Bc becomessmall. For example, in the case where Ymin and Ymax are 500 and 800,respectively, Ay is 3.41 and By is 1,705. Further, in the case whereYmin and Ymax are 750 and 800, respectively, Ay is 20.46 and By is15,345.

<Main Gamma Processing>

As the main gamma processing at step S310, the long-time gammaprocessing 408 performed for the long-time exposure image 43 and theshort-time gamma processing 409 performed for the short-time exposureimage 44 are explained in detail.

For the long-time exposure image 43, the long-time gamma processing 408is performed by expression (5) below. Here, it is assumed that the R, G,and B values of the input of the long-time exposure image 43 areR_(long), G_(long), and B_(long), respectively, and the R, G, and Bvalues after the gamma correction are R_(long)′, G_(long)′, andB_(long)′, respectively. Further, it is assumed that the luminance valueY and the color difference values U and V of the long-time exposureimage 43 are Y_(long), U_(long), and V_(long), respectively, and theluminance value Y and the color difference values U and V after thegamma correction are Y_(long)′, U_(long)′, and V_(long)′, respectively.Furthermore, Max and O_(max) are the same as those explained inexpression (1), i.e., the gamma correction coefficients set in advance.Still furthermore, the adjustment parameters Ay, By, Ac, and Bc are theparameters calculated at step S309.

                                expression  (5)Y_(long) = 0.257 * R_(long) + 0.504 * G_(long) + 0.114 * B_(long)U_(long) = −0.169 * R_(long) − 0.331 * G_(long) + 0.500 * B_(long)V_(long) = 0.500 * R_(long) − 0.419 * G_(long) − 0.081 * B_(long)$Y_{long}^{\prime} = {{{{Ay} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( Y_{long} \right)}} - {By}}$$U_{long}^{\prime} = {{{{Ac} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( U_{long} \right)}} - {Bc}}$$V_{long}^{\prime} = {{{{Ac} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( V_{long} \right)}} - {Bc}}$R_(long)^(′) = 1.00 * Y_(long)^(′) + 1.402 * V_(long)^(′)G_(long)^(′) = 1.00 * Y_(long)^(′) − 0.344 * U_(long)^(′) + 1.402 * V_(long)^(′)B_(long)^(′) = 1.00 * Y_(long)^(′) + 1.772 * U_(long)^(′)

Ay and Ac are parameters that can adjust the slope of the output valuefor the input value in the gamma processing. Specifically, as expressedin expression (5), as Ay becomes small, the slope of Y_(long)′ for log(Y_(long)) becomes small and as Ac becomes small, the slope of U_(long)′for log (U_(long)) and the slope of V_(long)′ for log (V_(long)) becomesmall. That is, as Ay and Ac become small, the slope of the output valuefor the input value becomes small in the gamma processing. Further, byexpression (4) described previously, in the case where the differencebetween Ymin and Ymax is large, Ay and Ac are calculated as smallvalues. Because of this, the main gamma processing of the presentembodiment becomes gamma processing adjusted so that the larger thedifference between Ymin and Ymax, the smaller the slope of the outputvalue for the input value becomes.

Further, By and Bc are parameters that can adjust the correctionconstant that decreases the output value on the whole in the gammaprocessing. Specifically, as expressed by expression (5), as By and Bcbecome small, the correction constant becomes large. Further, byexpression (4) described previously, in the case where Ymin is large, Byand Bc are calculated as large values. Because of this, the main gammaprocessing of the present embodiment becomes gamma processing adjustedso that the correction constant that decreases the output value on thewhole becomes large in the case where Ymin is large.

Further, for the short-time exposure image 44, the short-time gammaprocessing 409 is performed by expression (6) below. Here, it is assumedthat the R, G, and B values of the short-time exposure image 44 areR_(short), G_(short), and B_(short), respectively, and the R, G, and Bvalues after the gamma correction are R_(short)′, G_(short)′, andB_(short)′, respectively. Further, it is assumed that the luminancevalue Y and the color difference values U and V of the short-timeexposure image 44 are Y_(short), U_(short), and V_(short), respectively,and the luminance value Y and the color difference values U and V afterthe gamma correction are Y_(short)′, U_(short), and V_(short)′,respectively. Furthermore, it is assumed that the exposure time of theshort-time exposure image 44 is t_(short) and the exposure time of thelong-time exposure image 43 is t_(long).

                                expression  (6)Y_(short) = 0.257 * R_(short) + 0.504 * G_(short) + 0.114 * B_(short)U_(short) = −0.169 * R_(short) − 0.331 * G_(short) + 0.500 * B_(short)V_(short) = 0.500 * R_(short) − 0.419 * G_(short) − 0.081 * B_(short)$Y_{short}^{\prime} = {{{{Ay} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( {\frac{t_{long}}{t_{short}}Y_{short}} \right)}} - {By}}$$U_{short}^{\prime} = {{{{Ac} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( {\frac{t_{long}}{t_{short}}U_{short}} \right)}} - {Bc}}$$V_{short}^{\prime} = {{{{Ac} \cdot \frac{O_{\max}}{\log ({Max})}}{\log \left( {\frac{t_{long}}{t_{short}}V_{short}} \right)}} - {Bc}}$R_(short)^(′) = 1.00 * Y_(short)^(′) + 1.402 * V_(short)^(′)G_(short)^(′) = 1.00 * Y_(short)^(′) − 0.344 * U_(short)^(′) + 1.402 * V_(short)^(′)B_(short)^(′) = 1.00 * Y_(short)^(′) + 1.772 * U_(short)^(′)

As in the case of the long-time exposure image 43, the main gammaprocessing performed for the short-time exposure image 44 by expression(6) is gamma processing adjusted so that the larger the differencebetween Ymin and Ymax, the smaller the slope of the output value for theinput value becomes.

<HDR Combination Processing>

The main HDR combination processing at step S311 is explained. In thepreliminary HDR combination processing, images after the preliminarygamma processing are combined, but in the main HDR combinationprocessing, images after the main gamma processing are combined. Themain HDR combination processing is substantially the same as thepreliminary HDR combination processing, but the threshold value settingat the time of generating the composite coefficient map is different.Threshold values Th1′ and Th2′ in the main HDR combination processingare calculated from Th1 and Th2 in the preliminary HDR combinationprocessing as expression (7) below by using the adjustment parameters Ayand By.

Th1′=Ay·Th1−By

Th2′=Ay·Th2−By  expression (7)

As explained above, in the present embodiment, based on the results ofperforming the preliminary gamma processing and the preliminary HDRcombination processing for the long-time exposure image and theshort-time exposure image, the adjustment parameters to adapt the gammaprocessing to the dynamic range of a scene are calculated. The maingamma processing to which the calculated adjustment parameters have beenapplied is performed for the long-time exposure image and the short-timeexposure image, respectively, and the main HDR combination processing isperformed for the obtained long-time exposure image and short-timeexposure image after the gamma processing. As described above, byperforming the gamma processing adapted to the dynamic range of a sceneand the HDR combination of images after the gamma processing, it is madepossible to generate a combined image of high image quality even by thecombination processing with a small number of bits. Further, it is madepossible to perform the HDR combination adapted to the dynamic range ofa scene without the need to change the exposure conditions, such as acombination of exposure times for each scene.

In the above, the example is explained in which HDR combination isperformed for the two kinds of image with different exposure times, butin the present embodiment, it is also possible to perform HDRcombination of three or more kinds of image with different exposuretimes. At the time of performing HDR combination of three or more kindsof image with different exposure times, it is possible to performcombination in the order from the image with the longest exposure timeor with the shortest exposure time. For example, the case is explainedwhere three kinds of image with different exposure times are combined inthe order from the image with the longest exposure time. First, theimage with the longest exposure time and the image with the secondlongest exposure image are combined. The pixel value of the image, whichis the results of the combination, is taken to be Out. Then, asexpressed by expression (8) below, the image with the shortest exposuretime is further combined.

Out′(x,y)=A×I _(short)′(x,y)+(1−A)×Out(x,y)  expression (8)

Here, Out′ is the pixel value of the new image after the combination and‘short’ is the pixel value of the image with the shortest exposure timethat is newly combined with Out. The composite coefficient A inexpression (8) is generated based on the pixel value (here, I_(short)′)of the image with the shortest exposure time among the images to becombined.

In the present embodiment, explanation is given on the assumption thatthe image processing unit 209 is included in the image capturingapparatus 101, but the configuration is not limited to this and theimage processing unit 209 may be included in another apparatus or devicecapable of communicating with the image capturing apparatus 101.Alternatively, it may also be possible to configure the image processingunit 209 as an image processing apparatus capable of communicating withthe image capturing apparatus 101.

Second Embodiment

In the present embodiment, the case where the first embodiment isapplied to a moving image is explained. In the present embodiment, thepreliminary gamma processing and the preliminary HDR combination are notperformed and by using the adjustment parameters calculated based on theoutput image of the previous frame of the successive frames of a movingimage, the gamma characteristics applied to the main gamma processing ofthe current frame are adjusted. Explanation of the processing in commonto that of the first embodiment is omitted.

FIG. 7 is a diagram showing a flowchart of entire processing of theimage capturing apparatus 101 in the present embodiment. FIG. 8 is adiagram showing details of image processing performed by the imageprocessing unit 209 in the present embodiment.

At step S701, a user sets exposure conditions, such as the f-stop of thelens and the ISO speed, via the operation button 105. The exposure timeis not changed for each scene, but a condition prepared in advance isset.

At step S702, the control unit 206 determines whether or not the imagecapturing button 103 has been pressed down. In the case where the imagecapturing button 103 has been pressed down, the processing advances tostep S703 and in the case where the image capturing button 103 has notbeen pressed down, the processing returns to step S701.

At step S703, the control unit 206 sets the focus position by autofocusing and the exposure conditions (image capturing parameters) set bya user at step S701 to the image capturing system control unit 205 andinstructs the image capturing system control unit 205 to start the imagecapturing operation. Then, the image capturing system control unit 205captures a plurality of images with different exposure times (exposurebracket image capturing) based on the set exposure conditions andacquires a long-time exposure image and a short-time exposure image.Here, in the nth frame, a long-time exposure image 81 and a short-timeexposure image 82 are acquired. The acquired long-time exposure image 81and short-time exposure image 82 are input to the image processing unit209.

At step S704, the image processing unit 209 performs HDR combinationprocessing 801 for the long-time exposure image 81 and the short-timeexposure image 82 acquired at step S703 and generates an output image 83of the current frame. In the HDR combination processing 801 of thepresent embodiment, the same white balance processing as that at stepS304, the same demosaicking processing as that at step S305, the samemain gamma processing as that at step S310, and further, the same mainHDR combination processing as that at step S311 are performed. However,in the present embodiment, the preliminary gamma processing and thepreliminary HDR combination processing are not performed and by usingthe adjustment parameters calculated based on the output image(reference image) of the previous frame, the gamma characteristicsapplied to the main gamma processing of the current frame are adjusted.Here, adjustment parameters 80 are calculated in the (n−1)th frame basedon an output image of the frame. Further, the calculated adjustmentparameters 80 are held by the holding unit and are read from the holdingunit in the nth frame and used in the main gamma processing and the mainHDR combination processing for the long-time exposure image 81 and theshort-time exposure image 82. In the case immediately after the start ofthe processing, in order to perform the HDR combination processing forthe first frame, Ay=0, By=0, Ac=0, and Bc=0 are set.

At step S705, the image processing unit 209 performs adjustmentparameter calculation 802 based on the output image 83 of the currentframe generated at step S704. In the adjustment parameter calculation802 of the present embodiment, first, the same histogram calculation asthat at step 308 is performed. That is, for the output image 83 of thecurrent frame, the histograms of the luminance value Y and the R, G, andB values are calculated. Then, based on the calculated histograms of theoutput image 83 and the adjustment parameters 80 calculated in theprevious frame, adjustment parameters 84 are calculated by expression(9) below.

$\begin{matrix}\left\{ {\begin{matrix}{{{Ay}(n)} = \frac{M_{out}}{\frac{{Y\; \max} + {{By}\left( {n - 1} \right)}}{{Ay}\left( {n - 1} \right)} - \frac{{Y\; \min} + {{By}\left( {n - 1} \right)}}{{Ay}\left( {n - 1} \right)}}} \\{{{By}(n)} = {{{Ay}(n)} \cdot \frac{{Y\; \min} + {{By}\left( {n - 1} \right)}}{{Ay}\left( {n - 1} \right)}}}\end{matrix}\left\{ \begin{matrix}{{{Ac}(n)} = \frac{M_{out}}{\frac{{C\; \max} + {{Bc}\left( {n - 1} \right)}}{{Ac}\left( {n - 1} \right)} - \frac{{C\; \min} + {{Bc}\left( {n - 1} \right)}}{{Ac}\left( {n - 1} \right)}}} \\{{{Bc}(n)} = {{{Ac}(n)} \cdot \frac{{C\; \min} + {{Bc}\left( {n - 1} \right)}}{{Ac}\left( {n - 1} \right)}}} \\{{C\; \max} = {\max \left( {{R\; \max},{G\; \max},{B\; \max}} \right)}} \\{{C\; \min} = {\min \left( {{R\; \min},{G\; \min},{B\; \min}} \right)}}\end{matrix} \right.} \right. & {{expression}\mspace{14mu} (9)}\end{matrix}$

Here, Ay (n), By (n), Ac (n), and Bc (n) are the adjustment parameters84 calculated in the current frame and Ay (n−1), By (n−1), Ac (n−1), andBc (n−1) are the adjustment parameters 80 calculated in the previousframe. Further, Ymax, Ymin, Rmax, Rmin, Gmax, Gmin, Bmax, and Bmin arethe maximum values and the minimum values of Y, R, G, and B calculatedfrom the histograms of the output image 83 of the current frame.

At step S706, the image processing unit 209 outputs the output image 83generated at step S704 to the encoder unit 210, to the PC/media 213 viathe media I/F 211, and so on.

At step S707, the control unit 206 determines whether or not the endbutton has been pressed down. In the case where the end button has beenpressed down, the processing of the flow ends and in the other cases,the processing returns to step S703 and the processing at steps S703 toS707 is repeated.

As explained above, in the present embodiment, the adjustment parameterscalculated based on the output image of the previous frame of thesuccessive frames of a moving image are applied to the gamma processingand the HDR combination of the current frame. Due to this, it ispossible to perform the gamma correction and the HDR combination adaptedto the dynamic range of a scene. Further, at the time of capturingimages of a moving image also, by performing HDR combination of theimages after the gamma processing, it is made possible to generate acombined image of high image quality even by the combination processingwith a small number of bits. Furthermore, without the need to change thecombination of exposure times for each scene, it is made possible toperform the gamma processing and the HDR combination adapted to thedynamic range of a scene.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

It is possible for the image processing apparatus of the presentinvention to generate a combined image of high image quality by HDRcombination with a small number of bits without the need to prepare avery large number of parameters for scenes of a variety of dynamicranges.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2016-125786 filed Jun. 24, 2016, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An image processing apparatus that generates acombined image by performing gamma processing and combination processingfor a plurality of images obtained by capturing images of the same imagecapturing-target scene under different exposure conditions, the imageprocessing apparatus comprising: an acquisition unit configured toacquire adjustment parameters that adjust gamma characteristics appliedto the gamma processing in accordance with a dynamic range of the imagecapturing-target scene; a gamma processing unit configured to performthe gamma processing to which adjustment parameters acquired by theacquisition unit have been applied for the plurality of images; and acombination unit configured to generate the combined image by performingthe combination processing for the plurality of images for which thegamma processing has been performed by the gamma processing unit.
 2. Theimage processing apparatus according to claim 1, further comprising: apreliminary gamma processing unit configured to perform preliminarygamma processing based on predetermined gamma characteristics for theplurality of images; and a preliminary combination unit configured tocombine the plurality of images for which the preliminary gammaprocessing has been performed, wherein the acquisition unit calculates,by taking combination results by the preliminary combination unit as areference image, the adjustment parameters based on the reference image.3. The image processing apparatus according to claim 1, wherein theplurality of images is images captured in one frame of successive framesof a moving image, and the acquisition unit calculates, by taking thecombined image generated for the previous frame of the one frame as areference image, the adjustment parameters based on the reference image.4. The image processing apparatus according to claim 2, furthercomprising: a calculation unit configured to calculate a maximum valueand a minimum value of a pixel of the reference image, wherein theacquisition unit calculates the adjustment parameters based on a maximumvalue and a minimum value calculated by the calculation unit.
 5. Theimage processing apparatus according to claim 4, wherein the adjustmentparameters adjust the gamma characteristics so that the larger adifference between the maximum value and the minimum value, the smallera slope of an output value for an input value in the gamma processing ismade, and the larger the minimum value, the larger a correction constantthat reduces an output value on the whole in the gamma processing ismade.
 6. The image processing apparatus according to claim 4, whereinthe maximum value and the minimum value are calculated for each ofluminance, R, G, and B.
 7. The image processing apparatus according toclaim 4, wherein the calculation unit calculates histograms of thereference image and calculates the maximum value and the minimum valueby using calculated histograms.
 8. The image processing apparatusaccording to claim 1, wherein the combination unit changes a compositecoefficient of each image in the combination processing based on theadjustment parameters and performs combination processing in accordancewith a dynamic range of the image capturing-target scene.
 9. The imageprocessing apparatus according to claim 1, wherein the combinationprocessing is HDR combination.
 10. The image processing apparatusaccording to claim 1, wherein the plurality of images includes at leasta first image captured with a relatively short exposure time and asecond image captured with a relatively long exposure time.
 11. Theimage processing apparatus according to claim 1, wherein the gammaprocessing is log gamma correction on a YUV color space, and in a casewhere a pixel value in a YUV color space of the image is taken to be x,a pixel value in a YUV color space after gamma correction is taken to bey, and a coefficient set in advance is taken to be k, adjustmentparameters of the gamma processing are set as A and B in y=A·k·log(x)−B.12. An image processing method of generating a combined image byperforming gamma processing and combination processing for a pluralityof images obtained by capturing images of the same imagecapturing-target scene under different exposure conditions, the methodcomprising the steps of: acquiring adjustment parameters that adjustgamma characteristics applied to the gamma processing in accordance witha dynamic range of the image capturing-target scene; performing thegamma processing to which adjustment parameters acquired at theacquisition step have been applied for the plurality of images; andgenerating the combined image by performing the combination processingfor the plurality of images for which the gamma processing has beenperformed at the gamma processing step.
 13. A non-transitory computerreadable storage medium storing a program for causing a computer tofunction as an image processing apparatus that generates a combinedimage by performing gamma processing and combination processing for aplurality of images obtained by capturing images of the same imagecapturing-target scene under different exposure conditions, the imageprocessing apparatus comprising: an acquisition unit configured toacquire adjustment parameters that adjust gamma characteristics appliedto the gamma processing in accordance with a dynamic range of the imagecapturing-target scene; a gamma processing unit configured to performthe gamma processing to which adjustment parameters acquired by theacquisition unit have been applied for the plurality of images; and acombination unit configured to generate the combined image by performingthe combination processing for the plurality of images for which thegamma processing has been performed by the gamma processing unit.